PathogenAR
An augmented reality (AR) application providing interactive experiences with high-quality, scientifically accurate three-dimensional models related to infectious diseases.
PathogenAR is developed by the Office of Cyber Infrastructure and Computational Biology at the National Institute of Allergy and Infectious Diseases, part of the U.S. National Institutes of Health.
PathogenAR was conceptualized as a portable tool for the NIAID researchers and their collaborators to interactively communicate their work, with a strong emphasis on projects involving complex 3D Structures. Using immersive technologies such as AR are known to improve learning efficiency through increased understanding of content, learning spatial relationships, long-term memory retention, and increased student motivation.
This application has been designed to help researchers visually demonstrate complicated structural concepts to other researchers, while also providing an engaging interface for the public to explore the ongoing research at the NIAID and the scientific concepts behind that research.
Left: 3D Influenza Virion model from the "Influenza: Creation of a Universal Vaccine" module in PathogenAR.
Right: Stem Ferritin Nanoparticle Universal Vaccine candidate used in the "Influenza: Creation of a Universal Vaccine" module in PathogenAR.
Portable Communication Tool
3D printed models have become an important component of the NIAID communications with the public and the media, including recent high-profile demonstrations by the director of the NIAID, Dr. Anthony Fauci. In a practical sense, however, these physical objects are generally fragile, heavy, have limited resolution, and often lose varying degrees of accuracy when the models are adapted for 3D printing.
PathogenAR presents the same data at high resolution and in a portable format. By carrying these models on a mobile device they are available at a moment’s notice, whether at lunch with a colleague or for casting to a larger screen during a seminar or to highlight a concept for the media. With the COVID-19 crisis in play, users can share their screen during remote communications. The users can interactively focus on important or hard to reach areas of the model, the audience is then free to download the application themselves so they can later browse the underlying information at their convenience.
Above: The NIAID Vaccine Research Center Deputy Director, Dr. Barney Graham, using 3D prints of Influenza Hemagglutinin to explain vaccine creation at a Gates foundation event. While great communicative tools, these prints are large, heavy, fragile, and difficult to handle many at a time. (image credit - Mark Makela)
Below: The same Hemagglutinin model used in PathogenAR is interactable, projectable, lightweight, and can hold a large number of 3D models. The user can zoom in to see occluded spaces, highlight areas of interest (which also allows a simplified color scheme), and can toggle visualization types.
Choose Your Reality
PathogenAR has been designed with two distinct interactive modes, each with its own unique characteristics to engage the user.
Left: In MidAir Mode, users can place a 3D object directly into any space captured by the camera on their mobile device. They can then explore that model using tap-and-drag gestures on the screen or by physically moving ‘around’ the object. Tapping labels activates interactive educational content, including animations and text descriptions.
Right: With Merge Cube Mode, image targeting is used to lock the 3D object to a Merge Cube™, a commercially available patterned cube developed by Merge VR. This allows the user to ‘hold the model ‘in their hand’, providing an intuitive mechanism for rotating and zooming. (Marketing image, not real game play)
Creative Process
PathogenAR was created by a small team consisting of a UI/UX deisgner, several biochemistry subject matter experts, and a biomedical 3D modeling expert.
Our process starts with a research story which is timely, relevant to our institution, and suited to AR. The idea is storyboarded and we condense our talking points to limit text and maximize engagement. The researchers whose work we are featuring provide expert feedback before we start building.
The UI was designed to be simple, recognizable, and easy to use, keeping in mind that the user needs hold both a Merge Cube and the phone/tablet if in that mode. Colors used for highlighting have sufficient contrast for those with color-blindness. Each annotated scene has interactable components that bring up text descriptions, animations, and helpful definitions.
Sharing
All 3D assets created for PathogenAR are made freely available via the NIH 3D Print Exchange, a resource from the NIAID that hosts bioscientific and medical 3D models for download and free use. The models are accompanied by relevant metadata and source information.
Launch and Promotion
PathogenAR was released on June 2nd, 2020, with its first module describing the creation of a Universal Vaccine for Influenza Viruses. Promotional materials were distributed to the community within NIAID and more broadly within NIH (including the promotional video below), with a focus on soliciting feedback from those with an interest in VR/AR, structural biology, and vaccine research. Referral and download statistics are being monitored to identify where our outreach has been most effective.
A larger promotional campaign will correspond with the release of the next content modules currently in development. Social media, ListServs, professional associations, and academic institutions will be targeted more broadly. Due to the COVID-19 crisis, outreach at conferences and STEM events will not be possible this year. Instead, this time is being used to consolidate and integrate the feedback we are receiving, in anticipation of the next release.
PathogenAR Promotional Video
Moving Forward
PathogenAR has been designed with scalability in mind. It can grow and adapt to the stories our scientific community needs to tell. While several stories are in development, the current priority is COVID-19; researchers at the NIH and globally are working relentlessly to characterize the novel coronavirus, to identify effective treatments, and ultimately to synthesize vaccines to protect the population. The next release of PathogenAR will include a COVID-19 module, focused on communicating relevant scientific advances to the public.
Looking further forward, new modules for PathogenAR will leverage an even richer pool of multimedia and source data, including custom animations of drug target binding (HIV anti-retroviral activity) and CT scans (reconstruction of lungs from tuberculosis patients).
Left: 3D model of an HIV virion that will be featured in the next release of PathogenAR.
Right: 3D Model of SARS-CoV-2, the virus that causes COVID-19, that will be featured in the next release. The model was created by the NIAID biovisualization specialists and structural biologists based on electron micrographs and protein crystallography data.
